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ABSTRACT 

The goal of the Satellite Technology Demonstration 
(STD) is to create several types of educational television programs 
for a varied audience beamed by a communication satellite. The first 
program produced — "Time Out!" — was a career exploration series 
written for junior high students. A "formative" semester permitted 
the series to be shown over 16 weeks to one audience and then revised 
by an internal review panel. STD made the following recommer *\ ations: 
(1) In cases of limited resources and a diverse audience, evaluation 
by a review panel, supplemented with audience data, is more 
satisfactory than the use of selected audience members or 
consultants; (2) panel should be composed of 5-6 members possessing a 
variety of knowledge and responsibilities and should include those 
responsible for implementing the results; (3) evaluation should be 
initiated simultaneously with video-production development; (4) 
members should make independent ratings, but should share findings in 
group dialog; (5) whenever possible, the primary source of similar 
evaluations should be done by the original audience with a panel used 
as a supplementary source. (NR) 
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INTRODUCTION 



Tne Satellite Technology Demonstration (STD) has utilized the ATS-6 telecommunications 
satellite to distribute several types of educational television programs to varied audiences. 
The primary program, however, has been a career exploration series, titled "Time Out!", 
specifically designed for junior high school audiences. Because of its major role in the 
Demonstration, the "Time Out!" series has been the product of an extensive formative process 
which included a needs assessment, the field testing of sample scripts and video elements, 
and the establishment of a "formative" semester. The inclusion of a formative semester 
allowed the series to be shown over a sixteen-week period to one audience and subjected to 
revision prior to its reshowing to a second audience. Although the data inputs from overall 
formative process were varied and extensive, the STD decided to create an internal review 
panel. This paper will explore the contributions of this approach and will describe the 
methodology developed and the implementation of the process. 

WHEN IS THERE A NEED FOR AN INTERNAL REVIEW? 

The contribution of an internal review is ancillary to and determined by the scope and 
quality of the primary effort conducted with users. Formative efforts with users generally 
fall into two categories— either broad-based to obtain general acceptance data, or selective 
in-depth probes using a smaller subset of audience members. The broad-based approach may re- 
inforce a producer's positive or negative feelings about a product under development, but it 
seldom provides the detailed information needed to direct specific product modification. 
Similarly, the in-depth probe may provide detailed information, but its generality and val- 
idity are subject to question. When resources permit, organizations would be well advised 
to employ a combination of these two strategies. 

The Satellite Technology Demonstration (STD), because of scarce resources, was faced 
with choosing between the broad-based and selective in-depth approaches. However, in 
addressing the problem, it implemented what was perceived as a compromise al ternati ve--an 



internal review panel whose input could augment a broad-based acceptance effort conducted 
with primary users. The purpose of this panel was to provide constructive criticism in 
sufficient detail to facilitate program modifications. 

THE ORGANIZATION OF AN INTERNAL REVIEW 

The decision to implement an internal review panel was made by project management with 
the objective of providing an internal vehicle for monitoring program quality for subject 
matter content and production quality and assisting program revisions. The process involved 
the creation of a panel comprised of persons with both staff and administrative responsibili- 
ties within the project, including the associate project director, the director of research, 
the field coordinator, the content coordinator, and the executive producer. 

This group was assembled in the fall of 1974, subsequent to the completion of approxi- 
mately 60^ of the "Time Out!" series programs and the initiation of their broadcast during 
the first semester. In addition to the daily program ratings being prepared by students and 
teachers, production personnel required segment-specific information upon which to base 
effective modifications in the programs for an improved second semester broadcast version. 
For the information to have utility during the estimated remaining production schedule, the 
panel had a two-month period to complete its detailed ratings and to provide suggestions for 
changes . 

The nature of the information required for specific program modifications, in contrast 
to the information being collected from students and teachers, can be recognized when the 
structure of the programs is analyzed. Each 25-minute pretaped program consisted of several 
"segments" which were totally different in format and style. For instance, a single program 
might have four segments— one produced in "Time Control Central," a futuristic setting used 
for tying segments and program to daily instructional objectives; another in "Crossroads," 
a rural grocery store which included actor representation of youngsters of the Rocky Moun- 
tain Region student audience; a segment of "available materials," usually film clips on 
specific jobs; and a "Dr. DOT" segment, a carnival "barker" format used to present The Die- 



tionary of Occupational Titles. The type of questions asked (because of limited class time) 
of teachers and students revealed only that a total program had received poor acceptance by 
the audience. The information was not of sufficient detail to direct specific program modi- 
fications. The production staff needed to know which segment within a program was unaccept- 
able and, if possible, which specific "characteristics" of that segment were not appealing; 
i.e., lengthy discussion, redundancy of points being addressed, poor video quality, inappro- 
priate vocabulary. 

In add! -ion to the internal evaluation and the previously designed general acceptance 
data, a more dp**-'" " w d-semester instrument was designed and implemented for students and 
teachers. ■ produced user acceptance data on specific vehicles and formats, 

but dio ... -y:-z^-: . ze them with particular programs nor indicate variations in vehicle across 
prograr ;. <: origination of the in-house panel review and the more specific student eval- 
uation data were designed to provide the effective modifications of segments or entire pro- 
grams . 

The internal review was intended both to supplement and validate the student informa- 
tion. It was the panel's hope that the STD staff personnel would identify actual "character- 
istics" of segments which would be the cause of low student/teacher acceptance. It was 
assumed that instruments could be designed to obtain student acceptance on the "segment" 
level of programs, but that it was not efficiently possible to determine what particular for- 
mat or content characteristics generated the students' low acceptance. The panel review was 
designed to rate each segment in terms of specific characteristics as shown in Fig. 1 (edu- 
cational appropriateness, entertainment value, vocabulary level, acting believability and 
delivery, pacing, video and audio technical quality, etc.). This was the detailed informa- 
tion to be used and correlated with the student acceptance of entire segments, which would 
indicate the required segment change. 

IMPLEMENTATION OF PANEL ACTIVITIES 
Once its responsibilities had been identified, the panel met with content and production 
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Program 0-33 Reviewer 

Date of RevTevT „ 

I , P**ogram Objectives 

The student will identify careers that fall under or are related to the Farming, 
Fishery, and Forestry family. 

The student will Identify some viable options following graduation from high school. 
II. Point of Program 

To Introduce additional jobs that are Included in this job category and to point out 
required training, possibilities for advancement and the outlook for the future when 
feasible. 

To introduce options after high school. 

III. Segment Codes 

j207 - Ben Reviews FFF 11:30 

j208 - A Bit of Expert Advice 8:00 

j208a- Seriously, Now 1:30 
j55a - Eddie tries to Cheer up a Friend 5:30 
j56b - Ben and Sam Comment on Eddie 

Tries to Cheer up a Friend 1:30 

IV. (5) Excellent (4) Good (3) Average (2) Fair (1) Poor 

V. Segment Ratings 



Content j207 j208 j208a j55a J55b 

A. Educational Appropriateness 

B. Entertainment Value 

C. Vocabulary Leval 

0. Development of Ideas (Clarity, Seq.) 

Production 

A. Acting (Bel ievabi 1 1 ty , Delivery) 

B. Delivery Vehicle (Set, Special Effects, 

Format) ' 

C. Pacing (Segment Length, etc.) 

D. ■ Undorstandability (Diction, etc.) 

Technical Quality 

A. Video {Resolution, Color) 

B. Audio (Nci Level, Fluctuation) 

Segment in General 

A. General Rating 

R ecommendation 

A. Modify (y) yes, (n) no — 

VI. Comments/Suggestions - (Identify segment numbers) 

VII A. Identify any segments not appropriate for national audience. 

B. Identify any segments not appropriate for high school audiences. 



Figure 1. Evaluation form used 
by the Internal Review Team. 
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personnel. These meetings produced the instrument shown in Fig. 1. A two-month schedule 
for review of the programs was set up and the panel adopted a strategy of convening two 
afternoons per week to review six to eight -lf-hour programs per session. Reviewing con- 
secutive programs made comparisons between programs easier and enhanced the quantity of the 
panel's evaluation. However, it was soon found that reviewing more than five consecutive 

programs was too fatiguing. 

For the first three or four weeks after its implementation, the review panel met rou- 
tinely. As it became apparent that not all programs could be reviewed in the required time 
due to staff availability and a rigid evaluation schedule, more flexible scheduling was 1m- 
plemented-panel members could independently review films in accordance with their own 
schedules. This flexible schedule was maintained until the panel's activity was terminated 
because (1) essential trends on specific segments had been identified from the panel's and 
the previously referenced special student evaluation data, and (2) based on these trends, 
the remaining available production schedule had become committed to the necessary modifica- 
tions for entire formats such as the Dr. DOT segments. 

RESULTS OF THE INTERNAL REVIEW PANEL 

Although the panel die not review all programs, it provided meaningful data on 84% of 
the 69 pre-taped programs in the "Time Out!" series. The information yield from the review 

panel's efforts consisted of: 

(1) A trend line which illustrated the relative quality of each program as perceived by 
the panel. This trend line was augmented by placing an asterisk beside the numeral of any 
program having one or more segments perceived to be in need of revision. 

(2) A trend line for each program, illustrating the relative quality of each segment 
as perceived by the panel. Segments in need of revision were clearly identified. 

(3) An evaluation form for each program which showed the composite ratings of panel 

members on each program attribute. 

(4) A compilation of panel members' comments and suggestions regarding each program. 
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Fig. 2 below presents a section of the trend line produced by the ratings of the review 
panel. It was desirable to see how closely the perceptions of the panel members correlated 
with the ratings made by the junior high school audience; therefore, the panel's ratings 
have been superimposed on student ratings for the same programs. Although student ratings 
were obtained on several program aspects (e.g., interaction, support materials, in-class ac- 
tivities, etc.) for the purposes of ttie comparison illustrated in Fig. 2, a trend line 
reflecting the percentage of students rating each program "excellent" was used. Because of 
the large number of youngsters involved, STD researchers found this rating to be a sensitive 



and reliable acceptance indicator. 
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TREND LINE BASED ON REVIEW PANEL RATING 
» PROGRAM WITH ONE OR MORE SEGMENTS NEEDING REVISION 
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Figure 2. Trend lines indicating student acceptance and panel's evaluation 
across programs Jl through J31 (excludes 320, 325, and 330, which were live). 



A perusal of Fig. 2 will show a positive correlation between the students' and the 
panel's ratings. However, the review panel also provided detailed feedback which was not 
available from the students. 

In addition to providing trend lines across programs, the panel also developed trend 
lines to illustrate quality variations within programs. Fig. 3 illustrates the trend lines 
of three programs. The general ratings of all three of these programs are considered posi- 
tive; however, as the graph indicates, the general ratings of 3.6 on program J 12 was obtain- 
ed by averaging the ratings of both very strong and very weak segments. Therefore, while 
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none of the segments in programs J 10 and J 15 were recommended for changes, two segments 
(j73 and j 141) in program J12 were recommended for revision. 
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Figure 3. Trend lines rewaling panel's evaluation 
of segments within programs J 10, J 12, and J IS. 



By referring to the composite detailed evaluation form for program J 12 , content and 
production personnel were able to determine what characteristics of each of the segments 
were weak. Segment j73, for example, received "good" ratings in areas relating to technical 
quality, but very low ratings on the development of ideas, pacing and understandabi 1 ity. 
Written comments by panel members, which augmented the ratings on j73 included: "Basic ve- 
hicle and idea are valid, but moves too fast!" "Isn't easily understood— too long a segment 
and dialogue too fast." Segment J141 was rated quite differently; it received high ratings 
in the content area but low ratings on acting, understandability (diction), and audio quali- 
ty. The panel members' written comments on this segment included: "Eddie doesn't speak 
clearly." "Acting strained." "Re-do Eddie's lines!" 

Approximately 50^ of the programs reviewed by the panel included one or more segments 
recommended for revision. These recommendations, along with data from several other sources 
were utilized by the production personnel responsible for modifying programs and segments. 
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Because resource restrictions limited the number of possible revisions actually implemented, 
the following priorities served as guidelines in the merger of recommendations and budget/ 
time/facilities limitations: 

First Priority - Revise formats or characterizations that have negative impact across 

several programs. 

Second Priority - Revise programs that have generally low acceptance throughout. 

Third Priority - Revise programs that have one or more unique segments which are weak 
but also some which are strong. 

To illustrate the use of these priority guidelines, reference can be made again to the 
segment ratings of program J12 in Fig. 3, which was reasonably strong overall, having two 
weak segments. One of these segments, j73, dealt with the W. C Fields type character, "Dr. 
DOT." General comments received from the primary student audience indicated that Dr. DOT's 
"carnival barker" role was too verbal and contained too much fast-paced dialogue to be effec- 
tive. Because "Dr. DOT 11 was in several programs and was rated as ineffective by panel mem- 
bers, all "Dr. DOT" segments were modified, including j73. "Dr. DOT" represents an example 
of a first-priority item. The other weak segment in the J12 program, however, was left in- 
tact because its weakness in diction and auditory problems did not adversely affect the rest 
of the program. This segment, therefore, represents an item accorded third-priority status. 

RECOMMENDATIONS REGARDING THE USE 
OF INTERNAL REVIEW PANELS 

The modifications, specific to program J12, illustrate the general application of the 
data provided by the review panel. Panel output was always used in conjunction with data 
from the primary audience. Having implemented an internal review panel and having experi- 
enced its contributions as well as its frustrations, the STD makes the following recommenda- 
tions: 

(1) Generally, there are three options available to obtain detailed guidance for pro- 
gram modifications: a. intense observations and experimentation with selected audience 
members; b. the use of consultants; or c. the utilization of a review panel. With limited 
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resources and with a diverse audience, the STD implemented the third option, supported by 
general data from audience members. Although the STD did not review all programs, the panel 
generated meaningful data sufficient to support our assertion. that the process is viable. 

(2) The review panel is time-consuming and requires perceptions from different disci- 
plines. Careful consideration should therefore be given to composition of the panel in 
terms of staff availability. Ideally, a review panel would contain five or six members rep- 
resenting varying levels of responsibility and different disciplines, including production 
and project management personnel who are most likely to be implementing the derived recom- 
mendations to insure consistency with the organization's contractual responsibilities. 

(3) The review panel, although a time-consuming task, is a good source of detailed 
professional information. We would recommend that if a review panel effort is implemented, 
it be initiated simultaneously with video-production development, and that the necessary 
budget considerations be given to provide the resources of time, personnel, and equipment 
needed. 

(4) In the implementation of the internal review process, the STD found that both the 
quality of the reviews and the morale of the participants we,e higher when panel members 
met as a group; i.e., each member making an individual rating, but also having an opportun- 
ity to share perceptions with other members. 

(5) The use of an internal panel can serve to effectively supplement, with more speci- 
fic and production-related detail, alternative sources of information. However, the imple- 
mentation of such an effort should be considered only after initially considering a project's 
capability of obtaining similar detail from the primary audience. It is recommended that 
personnel responsible for production and evaluation activities work closely together in the 
planning phases of both activities to investigate this possibility for effective design of 
formative evaluation data. 



This report was produced with funding from the National Institute 
of Education. The views expressed do not necessarily reflect those 
of the National Institute of Education or the V. S. Department of 
Health/ Education, and Welfare. 
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